Web Survey Bibliography
Background: Traditional modes of survey data collection show decreasing response rates and increasing costs. Web surveys potentially provide a cost-effective alternative. Opt-in volunteer web panels are widely used for market research or opinion polling, but less for academic or government research because of concerns about their representativeness arising from the effect of self-selection bias. Various methods attempt to make web panel surveys more representative of the population. We compared results from four UK web surveys with Natsal-3, a national probability sample survey.
Methods: The four web surveys were done by three UK market research companies, each with large volunteer web panels. A shortened Natsal-3 questionnaire was included on four web surveys: two used basic demographic quotas and two were modified with variables correlated with key outcomes as additional quotas. Panel members aged 18–44 years, who were resident in Great Britain, and who met the criteria set for the quotas were eligible. Each company was asked to provide a sample of 2000 participants, which achieved between 2000 and 2099 participants, with data collection between May and July, 2012. After weighting for age and sex, comparisons were made with Natsal-3 for demographic characteristics, key behaviours, and opinions, to examine whether modified quotas improved the results. We calculated mean absolute odds ratios (ORs) and the percentage of variables that significantly differed from Natsal-3 at the 5% significance level. We used Z tests based on bootstrap standard errors to assess whether the modified quota surveys provided estimates closer to Natsal-3 than the basic quota surveys. We used generalised estimating equations to assess whether the two modified quota surveys and likewise the two basic quota surveys provided consistent estimates. Six demographic and 35 behaviour or opinion variables were compared, including reporting of same-sex experience and attraction, vaginal sex in the past month, number of heterosexual partners, and attendance to a sexually transmitted infection clinic.
Findings: All four web surveys gave different results from Natsal-3 for most of the variables, and overall the two surveys with modified quotas did not provide estimates that were closer to Natsal-3 than the basic quota surveys. Estimates for men from the web surveys differed more from Natsal-3 than estimates for women. For men, the mean absolute OR was 1·81 for basic and 1·64 for modified surveys, and for women the results were 1·43 for basic and 1·42 for modified surveys. The differences between the web surveys and Natsal-3 were numerically greater for questions asked face-to-face than those asked in self-completion format, but no formal statistical comparison was made. The two modified quota surveys seemed to differ from each other (ie, were inconsistent, p=0·07), as did the basic quota surveys (p<0·0001). Adjustment of web panel quota controls did not lead to much improvement according to the results of Z tests, although for men improvements were more evident than for women. Moreover, meeting the modified quotas proved difficult, and the quotas had to be relaxed in both cases.
Interpretation: When measuring sensitive sexual behaviours in the Great Britain population, volunteer web panel surveys provided significantly different estimates than a probability computer-assisted personal interview or computer-assisted self-interview survey, and modified quotas did not clearly improve estimation. At present, non-probability web panels are not an appropriate means of gathering data if accurate estimates of population prevalence are the intention, regardless of whether quotas are basic or more sophisticated. This finding is consistent with findings from similar research from the USA and other European countries.
Funding: Medical Research Council and the Wellcome Trust.
Homepage (Abstract)/ (Full text)
Further details
Web survey bibliography - Marketing/business (336)
- Achieving Strong Privacy in Online Survey; 2017; Zhou, Yo.; Zhou, Yi.; Chen, S.; Wu, S. S.
- Where, When, How and with What Do Panel Interviews Take Place and Is the Quality of Answers Affected...; 2017; Niebruegge, S.
- Is There a Future for Surveys; 2017; Miller, P. V.
- Mobile Research im Kontext der digitalen Transformation; 2017; Friedrich-Freksa, M.
- Virtual reality meets sensory research; 2017; Depoortere, L.
- Online customer journey analysis: a data science toolbox; 2017; Bonnay, D.
- Comparing Twitter and Online Panels for Survey Recruitment of E-Cigarette Users and Smokers; 2016; Guillory, J.; Kim, A.; Murphy, J.; Bradfield, B.; Nonnemaker, J.; Hsieh, Y. P.
- Statistical Design for Online Experiments Across Desktops, Tablets, Smartphones (and Maybe Wearable...; 2016; Qian, P.; Sadeghi, S.; Arora, N. K.
- FocusVision 2015 Annual MR Technology Report; 2016; Macer, T., Wilson, S.
- The Effects of a Delayed Incentive on Response Rates, Response Mode, Data Quality, and Sample Bias in...; 2016; McGonagle, K., Freedman, V. A.
- A look at the unique data-gathering process behind the Harvard Impact Study; 2016; Vitale, J.
- Are sliders too slick for surveys?; 2016; Buskirk, T. D.
- Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk; 2016; Berinsky, A.; Huber, G. A.; Lenz, G. S.
- Web-based versus Paper-based Survey Data: An Estimation of Road Users’ Value of Travel Time Savings...; 2016; Kato, H.; Sakashita, A.; Tsuchiya, Tak.
- An Examination of Opposing Responses on Duplicated Multi-Mode Survey Responses; 2016; Djangali, A.
- Scientific Surveys Based on Incomplete Sampling Frames and High Rates of Nonresponse; 2016; Fahimi, M.; Barlas, F. M.; Thomas, R. K.; Buttermore, N. R.
- Adapting Labour Force Survey questions from interviewer-administered modes for web self-completion in...; 2015; Betts, P.; Cubbon, B.
- Internet Panels, Professional Respondents, and Data Quality; 2015; Matthijsse, S.; De Leeuw, E. D.; Hox, J.
- Are they willing to use the web? First results of a possible switch from PAPI to CAPI/CAWI in an establishment...; 2015; Ellguth, P.; Kohaut, S.
- GreenBook Research Industry Trends Report; 2015; Murphy, L. (Ed.)
- The role of gamification in better accessing reality and hence increasing data validity ; 2015; Bailey, P.; Kernohan, H.; Pritchard, G.
- Rewarding the Truth; 2015; Puleston, J.
- Impact of raising awareness of respondents on the measurement quality in a web survey; 2015; Revilla, M.
- Email subject lines and response rates to invitations to participate in a web survey and a face-to-face...; 2015; Sappleton, N.; Lourenco, F.
- Can a non-probabilistic online panel achieve question quality similar to that of the European Social...; 2015; Revilla, M.; Saris, W. E.; Loewe, G.; Ochoa, C.
- Mode Effects in Mixed-Mode Economic Surveys: Insights from a Randomized Experiment; 2015; Hsu, J. W.; McFall, B. H.
- Web-based survey, calibration, and economic impact assessment of spending in nature based recreation; 2015; Paudel, K. P., Devkota, N., Gyawali, B.
- The Influence of Answer Box Format on Response Behavior on List-Style Open-Ended Questions; 2014; Keusch, F.
- Improving Survey Response Rates in Online Panels Effects of Low-Cost Incentives and Cost-Free Text Appeal...; 2014; Pedersen, M. J., Nielsen, C. V.
- Matrix versus paging designs in a brand attribution task; 2014; Conrad, F. G., McCullough, W., Nishimura, R.
- Internet-Based Surveys: Methodological Issues; 2014; Albaum, G., Brockett, P., Golden, L., Han, V., Roster, C. A., Smith, S. M., Wiley, J. B.
- Use of a Google Map Tool Embedded in an Internet Survey Instrument: Is it a Valid and Reliable Alternative...; 2014; Dasgupta, S., Vaughan, A. S., Kramer, M. R., Sanchez, T. H., Sullivan, P. S.
- Sequential or Simultaneous Multi-Mode? Results from Two Large Surveys of Electric Utility Consumers; 2014; Jackson, C., Ledoux, C.
- Targeting the bias – the impact of mass media attention on sample composition and representativeness...; 2014; Steinmetz, S., Oez, F., Tijdens, K. G.
- Exploring selection biases for developing countries - is the web a promising tool for data collection...; 2014; Tijdens, K. G., Steinmetz, S.
- Measuring the very long, fuzzy tail in the occupational distribution in web-surveys; 2014; Tijdens, K. G.
- Moving answers with the GyroScale: Using the mobile device’s gyroscope for market research purposes...; 2014; Luetters, H., Kraus, M., Westphal, D.
- Clicking vs. Dragging: Different Uses of the Mouse and Their Implications for Online Surveys; 2014; Sikkel, D., Steenbergen, R., Gras, S.
- Innovation for television research - online surveys via HbbTV. A new technology with fantastic opportunities...; 2014; Herche, J., Adler, M.
- Online mobile surveys in Italy: coverage and other methodological challenges; 2014; Poggio, T.
- How Sliders Bias Survey Data; 2013; Sellers, R.
- Survey Research Response Rates: Internet Technology vs. Snail Mail ; 2013; Lanier, P. A., Tanner, J. R., Totaro, M. W., Gradnigo, G.
- The impact of New Zealand's 2008 prohibition of piperazine-based party pills on young people'...; 2013; Sheridan, J., Dong, C. Y., Butler, R., Barnes, J.
- How well do volunteer web panel surveys measure sensitive behaviours in the general population, and...; 2013; Erens, B., Burkill, S., Copas, A., Couper, M. P., Conrad, F.
- Effects of Gamification on Participation and Data Quality in a Real-World Market Research Domain ; 2013; Cechanowicz, J., Gutwin, C., Brownell, B., Goodfellow, L.
- Ideal participants in online market research: Lessons from closed communities; 2013; Heinze, A., Ferneley, E., Child, P.
- Online, face-to-face and telephone surveys—Comparing different sampling methods in wine consumer...; 2013; Szolnoki, G., Hoffmann, D.
- Where does the Fair Trade price premium go? Confronting consumers' request with reality; 2013; Langen, N., Adenaeuer, L.
- Customer satisfaction in Web 2.0 and information technology development; 2013; Sharma, G., Baoku, L.
- Research staff and public engagement: a UK study; 2013; Davies, S.